Skip to content

Add Mask2Former#20792

Merged
alaradirik merged 112 commits intohuggingface:mainfrom
shivalikasingh95:add-mask2former
Jan 16, 2023
Merged

Add Mask2Former#20792
alaradirik merged 112 commits intohuggingface:mainfrom
shivalikasingh95:add-mask2former

Conversation

@alaradirik
Copy link
Contributor

@alaradirik alaradirik commented Dec 16, 2022 •
edited
Loading

What does this PR do?

Adds Mask2Former to transformers.
Original repo: https://github.com/facebookresearch/Mask2Former/
Paper: https://arxiv.org/abs/2112.01527

Co-authored with @shivalikasingh95

To Do:

  • Fix model tests (hidden state shapes, loading the config)
  • Test model, visualize outputs
  • Update model cards

Before submitting

  • This PR fixes a typo or improves the docs (you can dismiss the other checks if that's the case).
  • [X ] Did you read the contributor guideline,
    Pull Request section?
  • [X ] Was this discussed/approved via a Github issue or the forum? Please add a link
    to it if that's the case.
  • [X ] Did you make sure to update the documentation with your changes? Here are the
    documentation guidelines, and
    here are tips on formatting docstrings.
  • Did you write any new necessary tests?

shivalikasingh95 and others added 30 commits August 16, 2022 14:45
@shivalikasingh95
Mask2Former initial commit
fc47dac
@shivalikasingh95
Merge branch 'huggingface:main' into add-mask2former
120427f
@shivalikasingh95
Added Mask2FormerConfig and PixelDecoder changes
4c22175
@shivalikasingh95
Merge branch 'huggingface:main' into add-mask2former
ca5b5be
@shivalikasingh95
reverting maskformer documentation and adding new file for mask2former
0cb6c82
@shivalikasingh95
Merge branch 'huggingface:main' into add-mask2former
8377e96
@shivalikasingh95
Merge branch 'add-mask2former' of https://github.com/shivalikasingh95…
bb313c3 
…/transformers into add-mask2former
@shivalikasingh95 @alaradirik
Update src/transformers/models/auto/configuration_auto.py
842ff10 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update src/transformers/models/auto/configuration_auto.py
8801381 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update src/transformers/models/auto/feature_extraction_auto.py
cc40783 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update src/transformers/models/auto/modeling_auto.py
b20306f 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update src/transformers/models/auto/modeling_auto.py
14555d1 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update src/transformers/models/mask2former/__init__.py
b95f285 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95
updated from_backbone_decoder_pixel_decoder_configs() classmethod
7882692
@shivalikasingh95
Merge branch 'add-mask2former' of https://github.com/shivalikasingh95…
88fdb2a 
…/transformers into add-mask2former
@shivalikasingh95
added copied from statement for feature extractor
a32150b
@shivalikasingh95 @alaradirik
Update src/transformers/models/mask2former/modeling_mask2former.py
c6a34ea 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update src/transformers/models/auto/configuration_auto.py
eaa6646 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95
Merge branch 'huggingface:main' into add-mask2former
bba5a95
@shivalikasingh95 @alaradirik
Update docs/source/en/model_doc/mask2former.mdx
bc06738 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95 @alaradirik
Update docs/source/en/model_doc/mask2former.mdx
1710221 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95
1. Added Transformer Module changes for mask2former
b51edc3 
2Added Deformable Detr Encoder classes from deformable_detr Implementation for pixel Decoder
Fixed Pixel Decoder Implementation
@shivalikasingh95
Merge branch 'huggingface:main' into add-mask2former
d5b371b
@shivalikasingh95
Added loss calculation changes and fixes for Mask2FormerInstanceSegme…
f5ac59b 
…ntation
@shivalikasingh95
Merge branch 'main' of https://github.com/huggingface/transformers in…
f9a7bc6 
…to add-mask2former
@shivalikasingh95
imports related merge fix
410be54
@shivalikasingh95
bug fix
81c5fcd
@shivalikasingh95 @alaradirik
updated doc string for pixel level module output
0a42c8c 
Co-authored-by: Alara Dirik <8944735+alaradirik@users.noreply.github.com>
@shivalikasingh95
1. Restructured pixel decoder in modeling file
62fbc15 
2. Added checkpoint conversion script for mask2former
3. Updated feature extractor for instance segmentation post processing
4. Doc string updates
5. config file fixes
@alaradirik
update post process methods
600c48d
NielsRogge
NielsRogge approved these changes Jan 4, 2023
View reviewed changes
Copy link
Contributor

@NielsRogge NielsRogge left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks a lot for working on this model! 🙏 Left some final comments.

@alaradirik alaradirik mentioned this pull request Jan 4, 2023
Closed
3 tasks
@NielsRogge
Copy link
Contributor

NielsRogge commented Jan 4, 2023 •
edited by alaradirik
Loading

So it seems there are 2 todo's left:

  • leverage AutoImageProcessor instead of adding a new one
  • make sure slow integration tests of Donut and Swin are still passing, possibly using MaskFormerSwin as backbone

@NielsRogge NielsRogge changed the title Add Mask2Former to transformers Add Mask2Former Jan 4, 2023
@shivalikasingh95
Copy link
Contributor

shivalikasingh95 commented Jan 4, 2023 •
edited by alaradirik
Loading

So it seems there are 2 todo's left:

  • leverage AutoImageProcessor instead of adding a new one
  • make sure slow integration tests of Donut and Swin are still passing, possibly using MaskFormerSwin as backbone

Sure I'll connect with @alaradirik and we'll fix these shortly and update you.

@shivalikasingh95
Copy link
Contributor

shivalikasingh95 commented Jan 5, 2023

@NielsRogge Just wanted to update that backbone for Mask2Former has been switched to MaskFormerSwin.
Changes to modeling_swin.py and modeling_donut_swin.py have been reverted so slow integration tests of Donut and Swin are passing now.

Conversion of all 30 checkpoints from Mask2Former model zoo using swin backbone corresponding to all 4 datasets and segmentation tasks is done and are available on the Hub. I just need to update the model cards. Will finish that shortly too.

@NielsRogge
Copy link
Contributor

NielsRogge commented Jan 5, 2023 •
edited
Loading

Thank you!

I'm just wondering why the issue was occurring only on Swin-base on one specific dataset. It would definitely be nice to clear that up, does it have to do with the image resolution?

For instance for UperNet (at #20648) I was able to perfectly convert all checkpoints that leverage Swin-base by using our SwinBackbone. This one was ported from the mmsegmentation library whose Swin implementation is here. So it's a bit strange. Might it be that we were just "lucky" with UperNet and OneFormer?

@alaradirik alaradirik merged commit 2411f0e into huggingface:main Jan 16, 2023
@molbap molbap mentioned this pull request Jan 27, 2026
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@NielsRogge NielsRogge NielsRogge approved these changes

+3 more reviewers

@sgugger sgugger sgugger approved these changes

@praeclarumjj3 praeclarumjj3 praeclarumjj3 left review comments

@shivalikasingh95 shivalikasingh95 shivalikasingh95 left review comments

Reviewers whose approvals may not affect merge requirements

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

6 participants

@alaradirik @HuggingFaceDocBuilderDev @NielsRogge @shivalikasingh95 @sgugger @praeclarumjj3